A coreference resolution system is to cluster all mentions that refer to the same entity in a given context. All coreference resolution systems need to tackle two main tasks: one task is to detect all of the potential mentions, and the other is to learn the linking of an antecedent for each possible mention. In this paper, we propose a hybrid rule-neural coreference resolution system based on actor-critic learning, such that it can achieve better coreference performance by leveraging the advantages from both the heuristic rules and a neural conference model. This end-to-end system can also perform both mention detection and resolution by leveraging a joint training algorithm. We experiment on the BERT model to generate input span representations. Our model with the BERT span representation achieves the state-of-the-art performance among the models on the CoNLL-2012 Shared Task English Test Set.

The target of a coreference resolution system is to cluster all mentions that refer to the same entity in a given context. All coreference resolution systems need to solve two subtasks; one task is to detect all of the potential mentions, and the other is to learn the linking of an antecedent for each possible mention. In this paper, we propose a reinforcement learning actor-critic-based neural coreference resolution system, which can achieve both mention detection and mention clustering by leveraging an actor-critic deep reinforcement learning technique and a joint training algorithm. We experiment on the BERT model to generate different input span representations. Our model with the BERT span representation achieves the state-of-the-art performance among the models on the CoNLL-2012 Shared Task English Test Set.

Most recent semantic frame parsing systems for spoken language understanding (SLU) are designed based on recurrent neural networks. These systems display decent performance on benchmark SLU datasets such as ATIS or SNIPS, which contain short utterances with relatively simple patterns. However, the current semantic frame parsing models lack a mechanism to handle out-of-distribution (\emph{OOD}) patterns and out-of-vocabulary (\emph{OOV}) tokens. In this paper, we introduce a robust semantic frame parsing pipeline that can handle both \emph{OOD} patterns and \emph{OOV} tokens in conjunction with a new complex Twitter dataset that contains long tweets with more \emph{OOD} patterns and \emph{OOV} tokens. The new pipeline demonstrates much better results in comparison to state-of-the-art baseline SLU models on both the SNIPS dataset and the new Twitter dataset (Our new Twitter dataset can be downloaded from https://1drv.ms/u/s!AroHb-W6_OAlavK4begsDsMALfE?e=c8f2XX ). Finally, we also build an E2E application to demo the feasibility of our algorithm and show why it is useful in real application.

Singular value decomposition (SVD) is one of the most popular compression methods that approximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. The parameters of a trained neural network model may affect task performance unevenly, which suggests non-equal importance among the parameters. Compared to SVD, the decomposition method aware of parameter importance is the more practical choice in real cases. Unlike standard SVD, weighted value decomposition is a non-convex optimization problem that lacks a closed-form solution. We systematically investigated multiple optimization strategies to tackle the problem and examined our method by compressing Transformer-based language models. Further, we designed a metric to predict when the SVD may introduce a significant performance drop, for which our method can be a rescue strategy. The extensive evaluations demonstrate that our method can perform better than current SOTA methods in compressing Transformer-based language models.

将大型矩阵分配到小矩阵中是模型压缩的流行策略。奇异值分解（SVD）在这种压缩策略中起着至关重要的作用，近似具有较少参数的学习矩阵。但是，SVD最大程度地减少了平方误差以重建原始矩阵而不衡量参数的重要性，从而为那些影响任务准确性的人提供了更大的重建误差。换句话说，SVD的优化目标与受过训练的模型的任务准确性不符。我们通过引入Fisher信息来权衡影响模型预测的参数的重要性来分析此先前未开发的问题，进行观察并解决该问题。这个想法导致了我们的方法：Fisher加权SVD（FWSVD）。尽管我们方法的分解矩阵并没有导致较小的重建错误，但我们发现我们所得的任务准确性更接近原始模型的性能。我们使用基于变压器的语言模型进行分析，显示我们的加权SVD很大程度上减轻了不匹配的优化目标，并可以以更高的压缩率维持模型性能。我们的方法可以直接压缩特定于任务的模型，同时比需要昂贵的模型预训练的其他紧凑型模型策略更好。此外，对压缩模型的评估表明，我们的方法可以进一步降低9％至30％的参数，对任务准确性产生不大的影响。

域分类是自然语言理解（NLU）中的基本任务，通常需要快速住宿到新的新兴域。即使新模型可访问，此约束使其无法培育所有先前的域。大多数现有的持续学习方法患有低精度和性能波动，特别是当旧数据和新数据的分布显着不同时。事实上，关键的真实问题不是没有旧数据的，而是效率效率恢复模型与整个旧数据集。是否有可能利用一些旧数据来产生高精度并保持稳定的性能，同时在不引入额外的普通公共表？在本文中，我们提出了一个可在各种环境下稳定地产生高性能的文本数据的一个封路数据不断学习模型。具体地，我们利用Fisher信息选择可以“记录”原始模型的关键信息的示例。此外，提出了一种称为动态重量整合的新颖方案，以在恢复过程中启用自由的自由学习。广泛的实验表明基线患有波动的性能，因此在实践中无用。相反，我们建议的CCFI显着且始终如一地优于平均精度高达20％的最佳最新方法，CCFI的每个组件有效地贡献了整体性能。

诸如BERT的预先接受的语言模型在各种自然语言处理任务中显示出显着的效果。但是，这些模型通常包含数百万个参数，这可以防止它们在资源受限设备上实际部署。已知知识蒸馏，重量修剪和量化是模型压缩中的主要方向。然而，通过知识蒸馏获得的紧凑型模型即使对于相对小的压缩比也可能遭受显着的精度下降。另一方面，只有少数量化尝试专门用于自然语言处理任务。它们患有小的压缩比或较大的错误率，因为需要对超参数的手动设置，并且不支持微粒子组 - 方向量化。在本文中，我们提出了一种自动混合精密量化框架，设计用于伯特，其可以同时在亚组 - 明智的水平中进行量化和修剪。具体而言，我们所提出的方法利用可微分的神经结构搜索，搜索自动地分配每个子组中的参数的比例和精度，同时捕获冗余参数组。对BERT下游任务的广泛评估揭示了我们所提出的方法通过提供相同的模型尺寸来实现相同的性能。我们还通过将我们的解决方案与Ottherbert等正交方法相结合来展示获得极其轻量级模型的可行性。

跨域冷启动推荐是推荐系统越来越新兴的问题。现有的作品主要专注于解决跨域用户推荐或冷启动内容推荐。但是，当新域在早期发展时，它具有类似于源域的潜在用户，但互动较少。从源域中学习用户的偏好并将其转移到目标域中是至关重要的，特别是在具有有限用户反馈的新到达内容上。为了弥合这一差距，我们提出了一个自训练的跨域用户偏好学习（夫妻）框架，针对具有各种语义标签的冷启动推荐，例如视频的项目或视频类型。更具体地，我们考虑三个级别的偏好，包括用户历史，用户内容和用户组提供可靠的推荐。利用由域感知顺序模型表示的用户历史，将频率编码器应用于用于用户内容偏好学习的底层标记。然后，建议具有正交节点表示的分层存储器树以进一步概括域域的用户组偏好。整个框架以一种对比的方式更新，以先进先出（FIFO）队列获得更具独特的表示。两个数据集的广泛实验展示了用户和内容冷启动情况的夫妇效率。通过部署在线A / B一周测试，我们表明夫妇的点击率（CTR）优于淘宝应用程序的其他基线。现在该方法在线为跨域冷微视频推荐服务。

超越在分销数据上的测试上，在分销（OOD）检测中最近的普及方式增加了。最近尝试分类OOD数据介绍了接近和远远检测的概念。具体而言，先前作品在检测难度方面定义了OOD数据的特征。我们建议使用两种类型的分布换档来表征ood数据的频谱：协变速和概念转移，其中协变速转移对应于样式的变化，例如噪声和概念移位表示语义的变化。该表征揭示了对每种类型的敏感性对OOD数据的检测和置信校准是重要的。因此，我们调查了捕获对改善它们的每种类型数据集偏移和方法的敏感性的得分功能。为此，我们从理论上得出了两个分数函数，用于ood检测，协变速分数和概念换档分数，基于对均分数的kl分解，并提出了一种几何启发方法（几何奥丁）来改善ood检测在两个班次下，只有分发数据。另外，所提出的方法自然地导致表现力的后HOC校准函数，其在分配和分发数据中产生最先进的校准性能。我们是第一个提出一种跨越检测和校准以及不同类型的班次工作的方法的方法。查看https://sites.google.com/view/geometric-decomposition的project页面。

Deep neural networks have attained remarkable performance when applied to data that comes from the same distribution as that of the training set, but can significantly degrade otherwise. Therefore, detecting whether an example is out-of-distribution (OoD) is crucial to enable a system that can reject such samples or alert users. Recent works have made significant progress on OoD benchmarks consisting of small image datasets. However, many recent methods based on neural networks rely on training or tuning with both in-distribution and out-of-distribution data. The latter is generally hard to define a-priori, and its selection can easily bias the learning. We base our work on a popular method ODIN 1 [21], proposing two strategies for freeing it from the needs of tuning with OoD data, while improving its OoD detection performance. We specifically propose to decompose confidence scoring as well as a modified input pre-processing method. We show that both of these significantly help in detection performance. Our further analysis on a larger scale image dataset shows that the two types of distribution shifts, specifically semantic shift and non-semantic shift, present a significant difference in the difficulty of the problem, providing an analysis of when ODIN-like strategies do or do not work.